{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ec4edcdf-100b-4caf-b13b-fe316b257a93",
   "metadata": {},
   "source": [
    "## Benchmarking Results - Uptrain Evals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63f85da5-8a66-4561-8779-823f650c7ba4",
   "metadata": {},
   "source": [
    "**Overview**: In this notebook, we will compare different UpTrain stanard evals against human judgements. For every example, we have an eval score along with an explanation. Each example is assigned a score by a human for every eval. \n",
    "\n",
    "There are total 6 evals, we have covered in this notebook:\n",
    "- Context Relevance\n",
    "- Response Conciseness\n",
    "- Response Match\n",
    "- Factual Accuracy\n",
    "- Response Completeness with respect to Context\n",
    "- Response Relevance\n",
    "\n",
    "Each score has a value between 0 and 1. \n",
    "\n",
    "For our evaluations, we have used Financial QA dataset. The FiQA dataset has roughly 6,000 questions and 57,000 answers. Financial QA is hard because the vocabularies are context specific. In this experiment, we have randomly picked 30 questions and performed our evaluations on top of it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a8001ae4-fff3-4877-b099-d5330035ff41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import polars as pl \n",
    "import os\n",
    "import tempfile\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ed5eb8df-b715-42d5-8300-5649d6039bf0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "url = \"https://uptrain-assets.s3.ap-south-1.amazonaws.com/data/benchmark.jsonl\"\n",
    "dataset_path = os.path.join('./', \"benchmark.jsonl\")\n",
    "\n",
    "if not os.path.exists(dataset_path):\n",
    "    import httpx\n",
    "    r = httpx.get(url)\n",
    "    with open(dataset_path, \"wb\") as f:\n",
    "        f.write(r.content)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d77f2a95-520a-4573-b9a8-4cba7666437f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset = pl.read_ndjson(dataset_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a6c4a0bc-6691-4d4c-a2f1-9f4d48bb950c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of test cases:  30\n",
      "Couple of samples:  shape: (2, 10)\n",
      "┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐\n",
      "│ question  ┆ response  ┆ ground_tr ┆ context   ┆ … ┆ human_sco ┆ human_sco ┆ human_sco ┆ human_sc │\n",
      "│ ---       ┆ ---       ┆ uth       ┆ ---       ┆   ┆ re_respon ┆ re_respon ┆ re_factua ┆ ore_resp │\n",
      "│ str       ┆ str       ┆ ---       ┆ str       ┆   ┆ se_releva ┆ se_match  ┆ l_accurac ┆ onse_com │\n",
      "│           ┆           ┆ str       ┆           ┆   ┆ nce       ┆ ---       ┆ y         ┆ pletenes │\n",
      "│           ┆           ┆           ┆           ┆   ┆ ---       ┆ f64       ┆ ---       ┆ …        │\n",
      "│           ┆           ┆           ┆           ┆   ┆ f64       ┆           ┆ f64       ┆ ---      │\n",
      "│           ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆ f64      │\n",
      "╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡\n",
      "│ Do the    ┆ Based on  ┆ No. When  ┆ It really ┆ … ┆ 0.0       ┆ 0.2       ┆ 1.0       ┆ 0.0      │\n",
      "│ activitie ┆ the given ┆ you file  ┆ depends   ┆   ┆           ┆           ┆           ┆          │\n",
      "│ s of my   ┆ context,  ┆ your      ┆ on the    ┆   ┆           ┆           ┆           ┆          │\n",
      "│ LLC need… ┆ ther…     ┆ Articles  ┆ type of…  ┆   ┆           ┆           ┆           ┆          │\n",
      "│           ┆           ┆ …         ┆           ┆   ┆           ┆           ┆           ┆          │\n",
      "│ How much  ┆ Based on  ┆ On $4K/mo ┆ There is  ┆ … ┆ 0.4       ┆ 0.5       ┆ 1.0       ┆ 0.5      │\n",
      "│ house can ┆ the given ┆ gross     ┆ no simple ┆   ┆           ┆           ┆           ┆          │\n",
      "│ I afford, ┆ context,  ┆ about     ┆ way to    ┆   ┆           ┆           ┆           ┆          │\n",
      "│ wai…      ┆ it i…     ┆ $1000/mo  ┆ calcul…   ┆   ┆           ┆           ┆           ┆          │\n",
      "│           ┆           ┆ c…        ┆           ┆   ┆           ┆           ┆           ┆          │\n",
      "└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of test cases: \", len(dataset))\n",
    "print(\"Couple of samples: \", dataset[0:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4dc8a31a-328c-4081-a9b0-99b330769a6b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['question', 'response', 'ground_truth', 'context', 'human_score_context_relevance', 'human_score_response_conciseness', 'human_score_response_relevance', 'human_score_response_match', 'human_score_factual_accuracy', 'human_score_response_completeness_wrt_context']\n"
     ]
    }
   ],
   "source": [
    "print(dataset.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b0ff006-2b2a-4b0d-8253-d4671e46e4cb",
   "metadata": {},
   "source": [
    "## Running Evals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "473067c8-066d-46f9-bce0-f6c104a31a56",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from uptrain import APIClient, Evals, ResponseMatching\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "53b4d43b-71e4-4bb1-9158-fbad1d0f322d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2023-11-05 03:04:24.482\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.remote\u001b[0m:\u001b[36mlog_and_evaluate\u001b[0m:\u001b[36m455\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain server\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "UPTRAIN_API_KEY = \"up-*****************\" ## INSERT YOUR UPTRAIN KEY HERE\n",
    "\n",
    "client = APIClient(uptrain_api_key=UPTRAIN_API_KEY)\n",
    "\n",
    "\n",
    "\n",
    "results = client.log_and_evaluate(\n",
    "    project_name=\"benchmark\",\n",
    "    data=dataset,\n",
    "    checks=[\n",
    "           Evals.CONTEXT_RELEVANCE,\n",
    "           Evals.RESPONSE_CONCISENESS,\n",
    "           ResponseMatching(method='llm'),\n",
    "           Evals.RESPONSE_COMPLETENESS_WRT_CONTEXT,\n",
    "           Evals.FACTUAL_ACCURACY\n",
    "           Evals.RESPONSE_RELEVANCE \n",
    "           ]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5294e7a3-85e1-49c6-9996-1114a572b046",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset = pl.DataFrame(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12418139-e714-4bee-907d-39f612322649",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Context Relevance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "50bbe356-9d29-462c-a267-872a097462b7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Can I use my long position stocks as margin for my short sold stocks?\n",
      "Context: Yes, you call the broker and tell him to use those shares to deliver to the short position.\n",
      "Yes you can.  This is known as a short selling against the box.  In the old days, this was used to delay a taxable event.  You could lock in a gain without triggering a taxable event.  Any loss on one side of the box would be offset by a loss on the other side, and vice versa.  However, the IRS clamped down on this, and you will realize the gain on your long position as soon as you go short on the other side.  See http://www.investopedia.com/terms/s/sellagainstthebox.asp. As to how to initiate the short cover, just transfer the long position to the same account as your short position and make sure your broker covers the short.  Should be relatively easy.\n",
      "When you short a stock, you can lose an unlimited amount of money if the trade goes against you. If the shorted stock gaps up overnight you can lose more money than you have in your account. The best case is you make 100% if the stock goes to zero. And then you have margin fees on top of that. With long positions, it's the other way around. Your max loss is 100% and your gains are potentially unlimited.\n"
     ]
    }
   ],
   "source": [
    "print('Question:', dataset[16]['question'][0])\n",
    "print('Context:', dataset[16]['context'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "24626a3c-66b3-4896-b2fa-84c8f0821d30",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "1. The question is asking whether it is possible to use long position stocks as margin for short sold stocks.\n",
      "2. The extracted context mentions that it is possible to use long position stocks to deliver to the short position, which implies that they can be used as margin.\n",
      "3. The context also mentions that this practice is known as short selling against the box and provides some historical background on its use.\n",
      "4. It further explains that the IRS has clamped down on this practice and that the gain on the long position will be realized as soon as the short position is initiated.\n",
      "5. The context provides instructions on how to initiate the short cover by transferring the long position to the same account as the short position.\n",
      "6. The context also mentions the risks associated with short selling, such as the potential for unlimited losses if the trade goes against you.\n",
      "7. It contrasts this with long positions, where the maximum loss is 100% and the gains are potentially unlimited.\n",
      "\n",
      "Based on this reasoning, the extracted context can answer the given question completely. Therefore, the score is 1.0.\n"
     ]
    }
   ],
   "source": [
    "print(dataset[16]['score_context_relevance'][0])\n",
    "print(dataset[16]['explanation_context_relevance'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65f6e55a-a1d5-4a56-8ceb-fa74d57b3cd7",
   "metadata": {},
   "source": [
    "In this example, we can clearly see that the context is highly relevant to the question because of which it has a high context relevancy score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "21ecf30d-6fa2-4793-8da8-3a3121946260",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Can I deduct personal loans or use them as tax “write offs?”\n",
      "Context:  \"Basically, the money you pay in student loan interest is tax deductible, which means as far as the IRS is concerned, you didn't make that money. However, what that saves you on your taxes is a percentage of a percentage; you save the amount of your current marginal rate on the money you paid as interest. Simple example with made-up numbers: Let's say you had a student loan outstanding, and you were making payments of $150 monthly on it. Total payments to said loan in one tax year would be $1800. Of that amount, let's for the sake of argument say that half, $900, was interest. You get your 1098-E with that number on it, and reduce your taxable income by that amount. You're currently doing well, not outstanding but OK, so you're in the 25% tax bracket that most single middle-classers are in. So, your reduction in taxable income of $900 saves you the 25% that those 900 simoleons would have been taxed at, which is $225. So, all told, this loan is a net drain on your disposable income of $1,575, of which $675 is pure cost of capital; you never received a dollar in disbursements to match this amount you're paying, so it's money lost now in return for previous gains. 10 years later, you pay off the debt. Now that $1800 is yours to keep, and to pay full  taxes on. You pay $225 more in taxes (actually, because of amortization, the amount of additional taxes has been steadily increasing as the interest portion of the loan payments has reduced) but have the remaining $1575 in your pocket to do something else. While there is good debt and bad debt, debt is debt; whether deductible or not, the IRS will never credit your tax bill in the amount of interest owed (AFAIK; if someone knows of a loan whose interest is a credit instead of a deduction I'm all ears). So, the deduction on this loan reduces your cost of capital to an effective APR of 4.5%, and because it's a student loan and not a mortgage, you don't have to itemize so this is in effect a \"\"free\"\" deduction (even with an FHA mortgage allowing me to deduct interest, property taxes and PMI, and the residual medical costs after insurance of having our new baby, the $11,900 standard deduction for my wife and I was still the better deal this year). But, you're still losing 4.5% per year to interest. That's your break-even; if the money you could use to pay your debt could earn a better return than 4.5%, then invest it, but if not, pay off the loan. Right now, investments that could make you 4.5% are at the bottom edge of a steep increase in risk and variance, so if your expected ROI is close, I'd lean toward paying off the debt.\"\n",
      "If it is a business loan, the borrower would be able to claim a deduction for any interest paid on the loan and the lender would include the interest earned as part of their taxable income. You need to be careful on what you do and don't include as income. If the repayments made to you by the borrower in a year is $10,000 but only $8,000 of that is interest and the other $2,000 is part of the principal being returned to the lender, then you would only claim $8,000 as your income and the borrower would only claim $8,000 as a business deduction. Of course if it is interest only, then you and the borrower would use the full $10,000.\n"
     ]
    }
   ],
   "source": [
    "print('Question:', dataset[5]['question'][0])\n",
    "print('Context:', dataset[5]['context'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "b757b495-fbdd-4180-b80a-1c0454fc4f01",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n",
      "1. Read the question: \"Can I deduct personal loans or use them as tax 'write offs?'\"\n",
      "2. Read the extracted context: The extracted context discusses the tax deductibility of student loans and provides an example of how the deduction works. It also mentions that the deduction reduces the cost of capital but does not credit the tax bill in the amount of interest owed.\n",
      "3. Compare the question and the extracted context: The extracted context provides information about the tax deductibility of student loans, but it does not specifically address personal loans. Therefore, it does not directly answer the question.\n",
      "4. Determine the answer: The extracted context can provide some relevant information about the tax deductibility of loans, but it does not answer the question completely. Therefore, the correct choice is (B) The extracted context can give some relevant answer for the given question but can't answer it completely.\n",
      "5. Print the score: 0.5\n",
      "0.5\n"
     ]
    }
   ],
   "source": [
    "print(dataset[5]['score_context_relevance'][0])\n",
    "print(dataset[5]['explanation_context_relevance'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9e18c33-2d96-4b2c-a100-22f5eff6e96a",
   "metadata": {},
   "source": [
    "As we can see for this particular example, the context is not relevant to the question and thus, achieving low context relevancy score."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b34a102-195b-4e33-a68e-b5b89edf3d54",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Comparative Analysis against human judgements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "7ae70750-0977-47fc-b0db-2d489658b9ef",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.15"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "human_context_relevance_score = list(dataset['human_score_context_relevance'])\n",
    "llm_context_relevance_score = list(dataset['score_context_relevance'])\n",
    "np.mean(np.abs(np.array(human_context_relevance_score)- np.array(llm_context_relevance_score)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36f86c4d-0ce7-415a-9c7b-acb36e37dfd0",
   "metadata": {},
   "source": [
    "This shows that the 'context relevancy' metric is 85% accurate in comparison to human evaluations. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b18b5893-9a39-446e-8f1d-0607dc849291",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Response Conciseness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "099c6985-ebb8-462a-9ead-4949041e6afd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Should I worry too much about saving my 20% down before buying my first house?\n",
      "Response: Based on the information in the context, it seems that there are varying opinions on whether you should worry too much about saving your 20% down before buying your first house. Some factors to consider include the need to avoid paying private mortgage insurance (PMI) and having liquidity for emergency funds or other investments. Additionally, having funds set aside for repairs may be a good idea. It is also mentioned that the current low interest rates might make it more sensible to put money into a different investment vehicle and that closing costs should be taken into account. Ultimately, the decision should depend on your personal financial situation, including factors like your monthly payments, interest rates, retirement savings, other debt, and the possibility of changes in life circumstances.\n"
     ]
    }
   ],
   "source": [
    "print('Question:', dataset[12]['question'][0])\n",
    "print('Response:', dataset[12]['response'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "d791a33b-dce6-4344-bfaa-db943f70ed9b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "1. Read the question: \"Should I worry too much about saving my 20% down before buying my first house?\"\n",
      "2. Read the response: \"Based on the information in the context, it seems that there are varying opinions on whether you should worry too much about saving your 20% down before buying your first house. Some factors to consider include the need to avoid paying private mortgage insurance (PMI) and having liquidity for emergency funds or other investments. Additionally, having funds set aside for repairs may be a good idea. It is also mentioned that the current low interest rates might make it more sensible to put money into a different investment vehicle and that closing costs should be taken into account. Ultimately, the decision should depend on your personal financial situation, including factors like your monthly payments, interest rates, retirement savings, other debt, and the possibility of changes in life circumstances.\"\n",
      "3. Compare the response to the question and identify any additional irrelevant information.\n",
      "4. The response provides additional information that is relevant to the question. It discusses factors to consider such as avoiding PMI, having liquidity for emergency funds or other investments, funds for repairs, low interest rates, different investment options, closing costs, and personal financial situation.\n",
      "5. None of the additional information provided in the response is irrelevant to the question.\n",
      "6. Based on the above analysis, the generated answer has no additional irrelevant information.\n",
      "\n",
      "Score: 1.0\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "print(dataset[12]['score_response_conciseness'][0])\n",
    "print(dataset[12]['explanation_response_conciseness'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b7d6bf2-06b1-45b5-9fec-e906451e005e",
   "metadata": {},
   "source": [
    "In this case, we can clearly see that even though the response is big but it does not contain any irrelevant information, resulting in a high 'response conciseness' score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "9eeb097a-0011-493b-9600-52b5ec8bdce0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Can I use my long position stocks as margin for my short sold stocks?\n",
      "Response: Yes, you can use your long position stocks as margin for your short sold stocks. You would need to transfer the long position to the same account as your short position and make sure your broker covers the short. It is important to note that short selling comes with the risk of losing an unlimited amount of money if the trade goes against you.\n"
     ]
    }
   ],
   "source": [
    "print('Question:', dataset[16]['question'][0])\n",
    "print('Response:', dataset[16]['response'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "caeca096-67da-407a-b687-63eb9f2c0fc4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n",
      "1. Read the question: \"Can I use my long position stocks as margin for my short sold stocks?\"\n",
      "2. Read the response: \"Yes, you can use your long position stocks as margin for your short sold stocks. You would need to transfer the long position to the same account as your short position and make sure your broker covers the short. It is important to note that short selling comes with the risk of losing an unlimited amount of money if the trade goes against you.\"\n",
      "3. Compare the response to the question:\n",
      "   - The response directly answers the question by stating that it is possible to use long position stocks as margin for short sold stocks.\n",
      "4. Check for additional irrelevant information:\n",
      "   - The response provides additional information about transferring the long position to the same account as the short position and the risk of losing an unlimited amount of money in short selling.\n",
      "   - While this additional information may be useful, it is not directly relevant to the question of whether long position stocks can be used as margin for short sold stocks.\n",
      "5. Determine the score:\n",
      "   - The generated answer has a little additional irrelevant information.\n",
      "   - Score: 0.5\n",
      "\n",
      "0.5\n",
      "0.5\n"
     ]
    }
   ],
   "source": [
    "print(dataset[16]['score_response_conciseness'][0])\n",
    "print(dataset[16]['explanation_response_conciseness'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "039e5e82-244e-4ed7-b7e9-41238deb1b3b",
   "metadata": {},
   "source": [
    "In this particular example, the response does contain additional irrelevant information, resulting in a low 'response conciseness' score."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f709ebf-84e0-49ef-8344-9940295ec118",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Comparative Analysis against human judgements "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "4e4ab277-dd9a-4b29-9655-9cb5da4c694c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "human_response_conciseness_score = list(dataset['human_score_response_conciseness'])\n",
    "llm_response_conciseness_score = list(dataset['score_response_conciseness'])\n",
    "\n",
    "np.mean(np.abs(np.array(human_response_conciseness_score)- np.array(llm_response_conciseness_score)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28a4ea41-74e4-4169-a2aa-84bcfe2276e1",
   "metadata": {
    "tags": []
   },
   "source": [
    "This shows that the 'response conciseness' metric is 75% accurate in comparison to human evaluations. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77f557ee-dda5-4470-909f-000219f19632",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Response Match  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "3c0a0b78-0282-4e60-b4cf-98a43823a872",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response: \n",
      "Based on the information provided in the context, the state tax issues for a New Jersey resident with DC tax withheld and likely refunded are as follows:\n",
      "\n",
      "1. The individual may file an extension request with the NJ Division of Taxation until they receive an answer from DC regarding the refund. Alternatively, they can file without claiming the credit and amend their return later if DC refuses to refund.\n",
      "\n",
      "2. It is unlikely that DC will consider the individual as a resident for staying a couple of months in hotels over the year.\n",
      "\n",
      "\n",
      "Ground Truth: If you already filed the DC return, you can try and wait with filing the NJ return until you get the answer from DC.  You can file an extension request with the NJ division of taxation here. Or, you can file without claiming the credit, and worst case amend later and claim it if DC refuse to refund. I find it highly unlikely that DC will decide that a person staying for a couple of months over the year in hotels will count as a resident.\n"
     ]
    }
   ],
   "source": [
    "print('Response:', dataset[19]['response'][0])\n",
    "print('Ground Truth:', dataset[19]['ground_truth'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "97580676-87cd-42bb-b4b7-8057eb44f313",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "print(dataset[19]['score_response_match'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de313b59-f4ab-4949-b847-6eb5cfe9f3fe",
   "metadata": {},
   "source": [
    "In this specific example, response has the same meaning as given in the ground truth. The two sentences are different syntactically but has the same meaning. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "224eea34-da54-4f67-89d3-72a6f9d359d7",
   "metadata": {},
   "source": [
    "### Comparative Analysis against human judgements "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "ecac684f-5341-4bbe-8de2-535ea375d0f9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.23558952039597197"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "human_response_match_score = list(dataset['human_score_response_match'])\n",
    "llm_response_match_score = list(dataset['score_response_match'])\n",
    "\n",
    "np.mean(np.abs(np.array(human_response_match_score)- np.array(llm_response_match_score)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77729487-31b8-4b13-80e5-3271698def86",
   "metadata": {
    "tags": []
   },
   "source": [
    "This shows that the 'response matching' metric is 78% accurate in comparison to human evaluations. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30cc1220-c08a-4ace-889f-48e74c70b5ca",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Factual Accuracy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "eee911d5-c936-45db-80a9-4007836434b5",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Lend money at a rate linked to the prime rate\n",
      "Response: The prime rate is the interest rate at which banks lend to consumers and businesses with the highest credit ratings.\n",
      "Context: The prime rate is the interest rate banks use amongst themselves to lend money to each other only. It is used as the basis (sometimes) for what interest rate banks charge you.  The prime rate is based loosely on the Fed rate.  There is a committee that meets regularly to set this and other industry interest rates. http://en.wikipedia.org/wiki/Prime_rate I am not 100% positive the following is totally accurate The banks keep our deposits and pay us interest for doing so.  They are paying us interest because they take yours, mine and everybody elses deposits as a large lump sum and invest that money.  Sometimes as business loans, sometimes as mortgages and sometimes as credit card. The banks have a book of business that will be EXACTLY how much credit they have extended to everybody.  But they do not keep that amount of cash in the vaults, only some smaller percentage of that large amount.  When I use my credit card and they need to transfer money to amazon.com, if they don't happen to have enough cash that day, they will just borrow from another bank that does, and the interest rate they pay to do so is the prime rate.  Since they are paying interest on the money they borrow to pay the debt I charged because they told me my credit was worth so much (...???...) they charge me a little bit more than that. Hence your credit card or mortgage's APR being based on the prime rate. I THINK that is what they do  If I am wrong leave a comment and I will update, or the mods can.\n",
      "The prime rate is actually the rate at which banks lend to consumers/businesses with the highest credit ratings.  The bank rate (more commonly known as the overnight rate) is the interest rate at which banks can borrow funds from the Bank of Canada or from each other.\n",
      "No.  The WSJ prime rate is 4.25%, even the Fed prime rate is 1.75%, way above the 1.20% you'll be making from your savings account. If you are high worth individual with great credit history, the bank might give you a personal loan at 4.25%. They won't care what you do with it as long as they get their payments. If you are not that creditworthy, they'll ask for a collateral, you can mortgage your house for example. It ends up being the sames thing, you get your money and do what you want with it. If you can make more than the interest rate the bank gave you, great, you made profit. The bank however won't agree to lend you money at 0.6% (1/2 of the 1.2% APY your savings account will bring). Why would they when they can loan that at prime rate of 4.25%?? The closest you can get to something like this is if you are a hot-shot wall street money manager with track record of making big profits. In that case the bank might put some money in your fund for you to manage, but that's not something a regular person can do.\n"
     ]
    }
   ],
   "source": [
    "print('Question:',dataset[25]['question'][0])\n",
    "print('Response:',dataset[25]['response'][0])\n",
    "print('Context:',dataset[25]['context'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "69678cd9-5d4f-40c8-90ea-f623cce9a757",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n",
      "1. The prime rate is the interest rate at which banks lend to consumers and businesses.\n",
      "Argument for yes: The context mentions that the prime rate is the interest rate banks charge consumers and businesses for loans.\n",
      "Argument for no: The context states that the prime rate is the interest rate banks use amongst themselves to lend money to each other only, not to consumers and businesses.\n",
      "Judgement: no. Argument for no looks stronger.\n",
      "2. The prime rate is for borrowers with the highest credit ratings.\n",
      "Argument for yes: The context mentions that if you have a high credit rating, the bank might give you a personal loan at the prime rate.\n",
      "Argument for no: The context does not explicitly state that the prime rate is specifically for borrowers with the highest credit ratings.\n",
      "Judgement: no. Argument for no looks stronger.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(dataset[25]['score_factual_accuracy'][0])\n",
    "print(dataset[25]['explanation_factual_accuracy'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac896702-5ef0-46c1-a449-0fb427be9ada",
   "metadata": {},
   "source": [
    "In this example, we can clearly see that the response contains hallucinations because of which the 'factual accuracy' is low."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "5b7bab18-311a-4e2a-8e35-7aa47613b920",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Should I worry too much about saving my 20% down before buying my first house?\n",
      "Response: Based on the information in the context, it seems that there are varying opinions on whether you should worry too much about saving your 20% down before buying your first house. Some factors to consider include the need to avoid paying private mortgage insurance (PMI) and having liquidity for emergency funds or other investments. Additionally, having funds set aside for repairs may be a good idea. It is also mentioned that the current low interest rates might make it more sensible to put money into a different investment vehicle and that closing costs should be taken into account. Ultimately, the decision should depend on your personal financial situation, including factors like your monthly payments, interest rates, retirement savings, other debt, and the possibility of changes in life circumstances.\n",
      "Context: The primary reason to put 20% down on your home is to avoid paying PMI (private mortgage insurance).  Anyone who buys a house with a down-payment of under 20% is required to pay for this insurance (which protects the lender in case you default on your loan). PMI is what enables people to buy homes with as little as 3-5% down. I would recommend against paying more than 20%, because having liquidity for emergency funds, or other investments will give you the sort of flexibility that's good to have when the economy isn't so great.  Depending on whether the house you purchase is move-in ready or a fixer-upper, having funds set aside for repairs is a good idea as well.\n",
      "I am currently in the process of purchasing a house. I am only putting 5% down. I see that some are saying that the traditional 20% down is the way to go. I am a first time homebuyer, and unfortunately we no longer live in the world where 20% down is mandatory, which is part of the reason why housing prices are so high. I feel it is more important that you are comfortable with what your monthly payments are as well as being informed on how interest rates can change how much you owe each month. Right now interest rates are pretty low, and it would almost be silly to put 20% down on your home. It might make more sense to put money in different vehicle right now, if you have extra, as the global economy will likely pick up and until it does, interest rates will likely stay low. Just my 2 cents worth.  EDIT: I thought it would not be responsible of me not to mention that you should always have extra's saved for closing costs. They can be pricey, and if you are not informed of what they are, they can creep up on you.\n",
      "I'd stick with 20% down. Truth is - we don't know enough about you. Are you single and staying that way? How is your retirement savings doing? As others asked, any other debt? You can put 20% down, take a breath and see how it's going. I did just that, the 20%. We then had a baby, and 5 nanny-years to pay for. When she was gone, all that money went to the mortgage, and after refinancing (with no points no closing) we have 7 years to go. Just under 20 years beginning to end. During that time we've saved for college (just about fully funded) and for retirement (both with matched 401(k) accounts).  Remember, if you lose your job, a house with a lower mortgage means nothing when there's still the next payment due. But that cushion of cash can be handy.\n"
     ]
    }
   ],
   "source": [
    "print('Question:', dataset[12]['question'][0])\n",
    "print('Response:', dataset[12]['response'][0])\n",
    "print('Context:', dataset[12]['context'][0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "13808754-c356-4d13-bf11-dab905c1b28e",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "1. There are varying opinions on whether you should worry too much about saving your 20% down before buying your first house.\n",
      "Argument for yes: The context mentions that some people are saying that the traditional 20% down is the way to go, implying that there are varying opinions on this matter.\n",
      "Argument for no: No arguments.\n",
      "Judgement: yes. Argument for yes looks stronger.\n",
      "\n",
      "2. Factors to consider include the need to avoid paying private mortgage insurance (PMI) and having liquidity for emergency funds or other investments.\n",
      "Argument for yes: The context explicitly states that having liquidity for emergency funds or other investments is a factor to consider.\n",
      "Argument for no: No arguments.\n",
      "Judgement: yes. Argument for yes looks stronger.\n",
      "\n",
      "3. Having funds set aside for repairs may be a good idea.\n",
      "Argument for yes: The context mentions that depending on whether the house is move-in ready or a fixer-upper, having funds set aside for repairs is a good idea.\n",
      "Argument for no: No arguments.\n",
      "Judgement: yes. Argument for yes looks stronger.\n",
      "\n",
      "4. The current low interest rates might make it more sensible to put money into a different investment vehicle.\n",
      "Argument for yes: The context mentions that interest rates are low and it might make more sense to put money into a different investment vehicle.\n",
      "Argument for no: No arguments.\n",
      "Judgement: yes. Argument for yes looks stronger.\n",
      "\n",
      "5. The decision should depend on your personal financial situation, including factors like your monthly payments, interest rates, retirement savings, other debt, and the possibility of changes in life circumstances.\n",
      "Argument for yes: The context explicitly states that the decision should depend on personal financial situation and lists various factors to consider.\n",
      "Argument for no: No arguments.\n",
      "Judgement: yes. Argument for yes looks stronger.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(dataset[12]['score_factual_accuracy'][0])\n",
    "print(dataset[12]['explanation_factual_accuracy'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f5eaff4-2232-4394-a597-70473ceccb87",
   "metadata": {},
   "source": [
    "All the facts contained in the response are true and can be verified from the given context."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3202a4e1-6ca0-4fb4-a6aa-c9ddc3900968",
   "metadata": {},
   "source": [
    "### Comparative Analysis against human judgements "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "7888e84a-315f-4c93-8023-bed7f9d5f4f6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12333333333333334"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "human_factual_accuracy_score = list(dataset['human_score_factual_accuracy'])\n",
    "llm_factual_accuracy_score = list(dataset['score_factual_accuracy'])\n",
    "\n",
    "np.mean(np.abs(np.array(human_factual_accuracy_score)- np.array(llm_factual_accuracy_score)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e41066bd-d7c7-4795-a99d-87e65f5fd67b",
   "metadata": {},
   "source": [
    "This shows that the 'factual accuracy' metric is 88% accurate in comparison to human evaluations. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49d51a99-6e53-4f07-b14c-b74b78c78644",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Response Completeness with respect to Context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "d9025dcb-faff-46d6-9492-c3371fe11ab0",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Can I use my long position stocks as margin for my short sold stocks?\n",
      "Response: Yes, you can use your long position stocks as margin for your short sold stocks. You would need to transfer the long position to the same account as your short position and make sure your broker covers the short. It is important to note that short selling comes with the risk of losing an unlimited amount of money if the trade goes against you.\n",
      "Context: Yes, you call the broker and tell him to use those shares to deliver to the short position.\n",
      "Yes you can.  This is known as a short selling against the box.  In the old days, this was used to delay a taxable event.  You could lock in a gain without triggering a taxable event.  Any loss on one side of the box would be offset by a loss on the other side, and vice versa.  However, the IRS clamped down on this, and you will realize the gain on your long position as soon as you go short on the other side.  See http://www.investopedia.com/terms/s/sellagainstthebox.asp. As to how to initiate the short cover, just transfer the long position to the same account as your short position and make sure your broker covers the short.  Should be relatively easy.\n",
      "When you short a stock, you can lose an unlimited amount of money if the trade goes against you. If the shorted stock gaps up overnight you can lose more money than you have in your account. The best case is you make 100% if the stock goes to zero. And then you have margin fees on top of that. With long positions, it's the other way around. Your max loss is 100% and your gains are potentially unlimited.\n"
     ]
    }
   ],
   "source": [
    "print('Question:',dataset[16]['question'][0])\n",
    "print('Response:',dataset[16]['response'][0])\n",
    "print('Context:',dataset[16]['context'][0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "dffae908-3f9d-4eee-b40d-c5b4ae17cd03",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "1. The question asks if the user can use their long position stocks as margin for their short sold stocks.\n",
      "2. The context provides relevant information about short selling against the box and how to initiate the short cover.\n",
      "3. The generated response acknowledges that the user can use their long position stocks as margin for their short sold stocks.\n",
      "4. The generated response also includes the information from the context about transferring the long position to the same account as the short position and ensuring the broker covers the short.\n",
      "5. The generated response mentions the risk of losing an unlimited amount of money if the trade goes against the user, which is relevant information.\n",
      "\n",
      "The generated answer answers the given question adequately by extracting all the relevant information from the context and providing a clear response.\n",
      "\n",
      "1.0\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "print(dataset[16]['score_response_completeness_wrt_context'][0])\n",
    "print(dataset[16]['explanation_response_completeness_wrt_context'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4370d46d-76c3-4ecc-a2b3-3fc9229ce7a4",
   "metadata": {},
   "source": [
    "In this example, the response has all the adequate information from the given context to answer the question."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0923cdf8-45dc-42a6-ab36-d9866d824304",
   "metadata": {},
   "source": [
    "### Comparative Analysis against human judgements "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "af5cfdcb-c5c1-42c6-bb7a-cabaabf312ac",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.27374\n"
     ]
    }
   ],
   "source": [
    "human_response_completeness_wrt_context_score = list(dataset['human_score_response_completeness_wrt_context'])\n",
    "llm_response_completeness_wrt_context_score = list(dataset['score_response_completeness_wrt_context'])\n",
    "np.mean(np.abs(np.array(human_response_completeness_wrt_context_score)- np.array(llm_response_completeness_wrt_context_score)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eab342ef-1091-42c8-aa69-69338da7fbea",
   "metadata": {
    "tags": []
   },
   "source": [
    "This shows that the 'response completeness wrt context' metric is 73% accurate in comparison to human evaluations. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f8fd388-7865-4172-80f0-d2efef61e965",
   "metadata": {},
   "source": [
    "## Response Relevance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "b97afe9d-76da-4c80-b988-3021f999859b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: State tax issues for NJ resident with DC tax withheld, and likely refunded\n",
      "Conntext: \n",
      "Based on the information provided in the context, the state tax issues for a New Jersey resident with DC tax withheld and likely refunded are as follows:\n",
      "\n",
      "1. The individual may file an extension request with the NJ Division of Taxation until they receive an answer from DC regarding the refund. Alternatively, they can file without claiming the credit and amend their return later if DC refuses to refund.\n",
      "\n",
      "2. It is unlikely that DC will consider the individual as a resident for staying a couple of months in hotels over the year.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('Question:', dataset[19]['question'][0])\n",
    "print('Response:', dataset[19]['response'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "b4773069-a7c0-4fe3-88ce-2c5c7d142038",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "1. The question asks for the state tax issues for a New Jersey resident with DC tax withheld and likely refunded.\n",
      "2. The response provides two points related to the state tax issues for the given scenario.\n",
      "3. Both points in the response are directly relevant to the question and provide information about the actions the individual can take regarding their tax situation.\n",
      "4. There is no additional irrelevant information in the response.\n",
      "5. Therefore, the generated answer has no additional irrelevant information.\n",
      "\n",
      "Score: 1.0\n",
      "1.0\n",
      "1. The question asks for the state tax issues for a New Jersey resident with DC tax withheld and likely refunded.\n",
      "2. The response provides two state tax issues that are relevant to the situation described in the question.\n",
      "3. The first issue mentioned is that the individual may file an extension request with the NJ Division of Taxation until they receive an answer from DC regarding the refund. This addresses the potential delay in receiving the refund and provides a solution for the individual to handle their tax filing in the meantime.\n",
      "4. The second issue mentioned is that it is unlikely that DC will consider the individual as a resident for staying a couple of months in hotels over the year. This addresses the potential issue of the individual being considered a resident of DC and having to pay additional taxes.\n",
      "5. Both of these issues are directly related to the state tax implications for a New Jersey resident with DC tax withheld and likely refunded.\n",
      "6. Therefore, the generated answer answers the given question adequately.\n",
      "\n",
      "The score is 1.0\n"
     ]
    }
   ],
   "source": [
    "print(dataset[19]['score_response_relevance'][0])\n",
    "print(dataset[19]['explanation_response_relevance'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27db91ee-1a20-4556-a030-eff7b11758f5",
   "metadata": {},
   "source": [
    "In this example, the response does contain all the required information to answer this particular question. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71033fd5-15c3-4d5d-8b7f-62380d8b2c7a",
   "metadata": {},
   "source": [
    "### Comparative Analysis against human judgements "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "84fbe08f-978d-45dd-85f1-d9a9bfe38dfc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2622222222222222"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "human_response_relevance_score = list(dataset['human_score_response_relevance'])\n",
    "llm_response_relevance_score = list(dataset['score_response_relevance'])\n",
    "np.mean(np.abs(np.array(human_response_relevance_score)- np.array(llm_response_relevance_score)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66eb064d-3d4f-453d-9365-948bcc9be850",
   "metadata": {},
   "source": [
    "This shows that the 'response relevancy' metric is 74% accurate in comparison to human evaluations. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
